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ABSTRACT 

Objective: To assess the clinical relevance and 
newsworthiness of the UK National Institute for Health 
Research (NIHR) Health Technology Assessment (HTA) 
Programme funded reports. 
Study design: Retrospective cohort study. 
Setting: The cohort included 311 NIHR HTA 
Programme funded reports publishing in HTA in the 
period 1 January 2007-31 December 2012. The 
McMaster Online Rating of Evidence (MORE) system 
independently identified the clinical relevance and 
newsworthiness of NIHR HTA publications and non- 
NIHR HTA publications. The MORE system involves 
over 4000 physicians rating publications on a scale of 
relevance (the extent to which articles are relevant to 
practice) and a scale of newsworthiness (the extent to 
which articles contain news or something clinicians are 
unlikely to know). 

Main outcome measures: The proportion of reports 
published in HTA meeting MORE inclusion criteria and 
mean average relevance and newsworthiness ratings 
were calculated and compared with publications from 
the same studies publishing outside HTA and non- 
NIHR HTA funded publications. 
Results: 286/311 (92.0%) of NIHR HTA reports were 
assessed by MORE, of which 192 (67.1%) passed 
MORE criteria. The average clinical relevance rating for 
NIHR HTA reports was 5.48, statistically higher than the 
5.32 rating for non-NIHR HTA publications (mean 
differences 6, 95% CI 0.04 to 0.29, p=0.01). Average 
newsworthiness ratings were similar between NIHR HTA 
reports and non-NIHR HTA publications (4.75 and 4.70, 
respectively; mean difference=0.05, 95% CI -0.1 8 to 
0.07, p=0.402). NIHR HTA-funded original research 
reports were statistically higher for newsworthiness 
than reviews (5.05 compared with 4.64) (mean 
difference=0.41, 95% CI 0.18 to 0.64, p=0.001). 
Conclusions: Funding research of clinical relevance is 
important in maximising the value of research 
investment. The NIHR HTA Programme is successful in 
funding projects that generate outputs of clinical 
relevance. 



Strengths and limitations of this study 
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INTRODUCTION 

Commissioning new research in areas of 
high clinical relevance is an important part 
of minimising research waste. Chalmers and 



The strength of this study is that a rigorous 
extensive and independent approach to asses- 
sing the clinical relevance and newsworthiness 
of funded research provided by MORE was 
applied to the National Institute for Health 
Research (NIHR) Health Technology Assessment 
(HTA) reports. 
A limitation of this study was that the data were 
reliant on funded research teams self-reporting 
their publications. It is most likely that the 
number of publications is under-reported from 
NIHR HTA-funded studies. 
Raters contributing to McMaster Online Rating of 
Evidence are voluntary and are therefore likely to 
be interested in evidence-based medicine 
Therefore, they may not be representative of all 
physicians. 



Glasziou 1 in their paper on avoidable waste 
in research argue that selecting unimportant 
research questions is a major contributor to 
wasted research funds. Studies have shown, 
for example, that only moderate correlations 
exist between publicly funded research and 
burden of disease. 2 Similarly, research con- 
ducted on patient or clinician research prior- 
ities has revealed potential mismatches 
between clinical priorities and funded 
research. Tallon et al, 3 for example, exam- 
ined the research priorities of patients with 
osteoarthritis of the knee and their clinicians 
and reported that their priorities differed 
from randomised controlled trials, which 
were typically focused on drug evaluations. 

One proxy measure of the clinical rele- 
vance of funded research is the assessment 
of the relevance of the final publication. 
McMaster University has established a rigor- 
ous approach to assessing the clinical rele- 
vance and newsworthiness of outputs from 
health research. 4 The McMaster approach 
involves selecting relevant papers from 120 
high-impact journals selected on the basis of 
suggestions from librarians, clinicians, editors 
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and editorial staff, Science Citations Index impact 
factors, a systematic examination of journal contents and 
an ongoing assessment of the number of articles that 
meet inclusion criteria of quality. Articles from selected 
journals are assessed by trained staff using inclusion cri- 
teria on methodological quality and whether the article 
has at least one clinically important outcome. 4 Articles 
passing these criteria are entered onto the Critical 
Appraisal Process (CAP) system before being transferred 
to the McMaster Online Rating of Evidence (MORE) 
system, which has in excess of 4000 physician raters in 
over 60 physician disciplines. Articles are sent to at least 
four raters for each matched discipline/specialty, who 
rate papers on two seven-point scales (seven — highest 
score; table 1). The first scale of relevance assesses the 
extent to which the article is relevant to practice. If rele- 
vance is rated three or above, a second scale is used 
assessing the extent to which the article contains news 
or something that clinicians are unlikely to know. 
Articles that average three or above for a discipline are 
then transferred to a Premium LiteratUre Service 
(PLUS) database, and are then used to alert subscribers 
via Evidence Updates, ACP JournalWise and other 
current awareness services. Ratings, which are collected 
at the time of journal article publication, have been 
shown to be predictive of subsequent citation counts. 5 

The McMaster approach to assessing clinical relevance 
and newsworthiness has enabled interesting comparisons 
to be made between research and publication types. 
One study explored the relevance and newsworthiness 
of high-quality original articles and systematic reviews 
from over 110 clinical journals and the Cochrane 
Database of Systematic Reviews. 6 The study revealed that 
systematic reviews had a significantly higher rating for 
relevance than original articles, but a significantly lower 
rating for newsworthiness. 6 In addition, clinicians 
accessed reviews more frequently than original articles. 6 



The National Institute for Health Research (NIHR) 
Health Technology Assessment (HTA) Programme was 
established in 1993 to commission research that is useful 
to clinical practice and NHS decision-makers. 7 Since its 
inception, significant effort has been made to identify and 
prioritise important topics from a range of stakeholders 
and to assess applications on the basis of scientific merit, 
feasibility and value for money. In the 10 years since 2002, 
98% of NIHR HTA-funded studies have completed with a 
final report in the HTA Journal 8 and the journal has always 
been included in the McMaster's top 120 journals. The 
reports (also referred to as monographs) are peer 
reviewed, are available freely in the public domain and 
contain a full record of the study. Unlike typical peer 
reviewed journals, the HTA Journal has a large word guide 
of approximately 50 000 words and unlimited appendices, 
thus enabling more detail to be included than standard 
peer review publications. Given the emphasis on identify- 
ing and commissioning important topics and having easily 
accessible and usable final reports, this study aimed to 
assess the clinical relevance and newsworthiness of NIHR 
HTA reports compared with non-NIHR HTA publications 
assessed by the MORE process. 

METHODS 

Selection of reports and publications 

Of 314 studies published as full reports in the NIHR 
HTA Journal in the period 1 January 2007—31 
December 2012, 311 were selected for the study. Three 
NIHR HTA reports were excluded from the analysis as 
they were updates of earlier Cochrane reviews, and 
therefore ratings of relevance and newsworthiness of 
these reports were not based on the first time these 
reviews were made available to the public. PubMed 
Identification numbers were identified for each report, 
which were obtained by hand searching PubMed. 



Table 1 


The McMaster Online Rating of Evidence (MORE) criteria for clinical relevance and newsworthiness 




Relevance to clinical practice in your 


Newsworthiness to clinical practice in your discipline — 


Score 


discipline — criteria 


criteria 


No score 


Beyond my personal area of expertise but 


I don't know if this is newsworthy 




may be of interest to my discipline 




7 


Directly and highly relevant 


Useful information; most practitioners in my discipline definitely do 






not know this (unless they have read this article) 


6 


Definitely relevant 


Useful information; most practitioners in my discipline probably do 






not know this 


5 


Probably relevant 


Useful information; most practitioners in my discipline possibly do 






not know this 


4 


Possibly relevant: likely of indirect or 


Useful information; most practitioners in my discipline possibly 




peripheral relevance at best 


already know this 


3 


Possibly not relevant 


Useful information; most practitioners in my discipline probably 






already know this 


2 


Probably not relevant: content only remotely 


It probably does not matter whether they know this or not 




related 




1 


Definitely not relevant: completely unrelated 


Not of direct clinical interest 




content area 







2 
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Publications arising from NIHR HTA-funded studies 
publishing outside the HTA Journal in the period 1 
January 2007-31 December 2012 were identified using 
research management databases maintained by the 
NIHR HTA Programme. Again, PubMed Identification 
numbers were obtained for each publication and these 
data were used to remove NIHR HTA Programme- 
funded research outputs from the database of articles 
from the 120 selected journals that had passed CAP and 
MORE criteria. A subgroup was identified for publica- 
tions from studies reporting in the HTA Journal in the 
period 2007-2012. This enabled a direct comparison 
between reports in the HTA Journal and related publica- 
tions from the same studies publishing outside the 
Journal. 

All data were assessed by a second researcher to 
ensure that there were no duplications and that the pub- 
lication and PubMed details were accurate. Data were 
held in an Excel spreadsheet, which was given to 
McMaster. The PubMed identification numbers were 
used to determine which articles were reviewed for and 
passed the CAP process, and then to automatically 
retrieve relevance and newsworthiness ratings from the 
MORE database. 

Units of analysis, hypotheses and statistical analyses 

Three datasets were generated that formed the basis of the 
analysis 

A. NIHR HTA-funded studies publishing as full reports 
in HTA in the period 1 January 2007-31 December 
2012. 

B. Publications in the period 1 January 2007-31 
December 2012 from NIHR HTA-funded studies pub- 
lishing outside HTA for those studies included in cat- 
egory A above. 

C. Publications from non-NIHR HTA-funded publica- 
tions publishing in the period 1 January 2007-31 
December 2012 from Journals included by McMaster 
and passing CAP and MORE criteria. 

Two principal hypotheses were tested 

Hypothesis 1: Reports published in HTA have a higher 
MORE clinical relevance and newsworthiness rating than 
non-NIHR HTA publications. The rationale for this 
hypothesis is that the NIHR HTA Programme aims to 
ensure that funded research responds to clinical need 
and has a rigorous process of identification and prioritisa- 
tion to reflect clinical priorities. We therefore wanted to 
see if outputs from NIHR HTA funded research rated 
higher in terms of clinical relevance and newsworthiness. 

Hypothesis 2: Reports published in HTA have a lower 
mean average MORE clinical relevance and newsworthi- 
ness rating than publications from the same research 
published outside the HTA Journal. The rationale for 
this is that funded research teams are encouraged to 
publish outside the HTA Journal, and therefore prior 
publication may affect the relevance ratings of the 
report if study findings are already in the public 



domain. Previous research had reported that some 
journal publications did precede publication in the HTA 
Journal, although it was recognised that this was not 
always the case and that for some studies, non-HTA 
Journal publications occurred well after the end of the 
project. 9 

Subgroup analysis was undertaken for hypotheses 
exploring differences in clinical relevance and news- 
worthiness ratings between original studies and review 
articles. Original studies are defined as 'Any full text 
article in which the authors report first-hand observa- 
tions'. Review articles are defined as 'any full text article 
that was bannered "review, overview, or meta-analysis" in 
the title or in a section heading, or it was indicated in 
the text of the article that the intention was to review, 
summarise, or highlight the literature on a particular 
topic'. 10 Previous research had identified that original 
research studies were more newsworthy but less clinically 
relevant than review articles. 6 We wished to explore 
whether this was also the case for NIHR HTA reports. 

As discussed elsewhere, 6 article ratings were treated as 
continuous variables and were analysed by calculating 
mean averages and proportions and comparing CIs of 
computed results. SPSS V.20 was used for all analyses; 
differences were compared using p values. 

RESULTS 

Proportions of reports and publications meeting CAP 
and MORE criteria 

Of 311 NIHR HTA reports published in the period 
2007-2012, 286 (92%) were included in the initial CAP 
assessment. Twenty-five reports were not included in the 
analysis due to a technical error, 18 of which were pub- 
lished in the period 20 February 2012-8 May 2012. Of 
the 274 HTA publications publishing outside the HTA 
Journal in the period 2007—2012, 118 (43.1%) were 
assessed for CAP. Non-inclusion of NIHR HTA-funded 
publications was due to the publications appearing in 
journals not included in the 120 journals assessed by 
McMaster. 

Table 2 reveals that 67.8% (n=194) of NIHR HTA 
reports passed initial screening criteria of methodo- 
logical rigour and relevance, compared with 57.6% 
(n=68) of other NIHR HTA-funded publications and 
10.6% (n=20 194) of non-NIHR HTA publications 
included on the McMaster database. A total of 67.1% 
(n=192) of NIHR HTA reports passed MORE inclusion 
(ie, scoring > 3/7 for both relevance and newsworthiness 
for at least one discipline), compared with 57.6% (n=68) 
of other NIHR HTA-funded publications and 10.5% 
(n=19 921) of non-NIHR HTA-funded publications. 

Clinical relevance ratings in MORE 

NIHR HTA reports had an average mean relevance 
rating of 5.48 (95% CI 5.36 to 5.59), which was higher 
than the mean relevance rating for non-NIHR HTA pub- 
lications passing CAP and MORE criteria, which was 
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Table 2 Proportion of publications passing CAP and MORE ratings of relevance and newsworthiness 


A: number (%) 
NIHR HTA reports 
2007-2012 


B: number (%) Publications 
2007-2012 from NIHR HTA 
studies included in A 


C: number (%) non-NIHR 
HTA publications 
2007-2012 


All publications 286 
Number (%) passing CAP 1 94 (67.8) 
Number (%) passing MORE in at 192 (67.1) 
least one discipline 


118 

68 (57.6) 
68 (57.6) 


190 391* 
20 194 (10.6) 
19 921 (10.5) 


'Estimated on PubMed searches of the number of articles published in all read journals, 2007-2012 (excluding editorials, letters). 

CAP, critical appraisal process; HTA, health technology assessment; MORE, McMaster Online Rating of Evidence; NIHR, National Institute 

for Health Research. 



5.32 (95% CI 5.31 to 5.33) (mean difference^. 16, 95% 
CI 0.04 to 0.29, p=0.01; table 3). NIHR HTA-funded 
publications publishing outside HTA had an average 
mean relevance rating of 5.59 (95% CI 5.42 to 5.75), 
which was slightly higher than the rating for NIHR HTA 
reports (mean difference=0.11, 95% CI -0.10 to 0.32, 
p=0.419). 

The mean relevance rating of NIHR HTA original 
research reports was 5.54 (95% CI 5.34 to 5.74), which 
was higher than non — NIHR HTA original research pub- 
lications, which had a mean relevance rating of 5.27 
(95% CI 5.26 to 5.29) (mean difference=0.27, 95% CI 
0.03 to 0.51, p=0.03; table 4). The mean relevance rating 
of NIHR HTA original research reports was similar to 
publications from the same studies publishing outside 
the HTA Journal, which had a mean average relevance 
rating of 5.57 (95% CI 5.36 to 5.78) (mean differ- 
ence=0.03, 95% CI -0.32 to 0.26, p=0.875; table 4). 

The mean relevance rating of NIHR HTA-funded 
review reports was 5.46 (95% CI 5.32 to 5.60), which was 
similar to that for non-NIHR HTA-funded publications, 
which had a mean relevance rating of 5.41 (95% CI 5.39 
to 5.43) (mean difference=0.05, 95% CI -0.09 to 0.19, 
p=0.46; table 5). NIHR HTA-funded review reports had 
a slightly lower mean relevance rating than publications 
from the same studies publishing outside the HTA 
Journal, which had a mean relevance rating of 5.62 
(95% CI 5.38 to 5.87) (mean difference =0.16, 95% CI 
-0.052 to 0.20, p=0.41; table 5). 

NIHR HTA-funded original research reports had a 
similar mean average relevance rating to NIHR HTA- 
funded review reports with a mean rating of 5.54 (95% CI 
5.34 to 5.74) compared with 5.46 (95% CI 5.32 to 5.60) 
(mean difference=0.08, 95% CI -0.33 to 0.17, p=0.54). 



Newsworthiness ratings in MORE 

NIHR HTA-funded reports had a mean newsworthiness 
rating of 4.75 (95% CI 4.60 to 4.86), which was similar 
to that for non-NIHR HTA publications, which had a 
mean newsworthiness rating of 4.70 (95% CI 4.69 to 
4.71; mean difference =0.05, 95% CI -0.18 to 0.07, 
p=0.402; table 3). NIHR HTA-funded publications pub- 
lishing outside the HTA Journal had a higher average 
mean newsworthiness rating than NIHR HTA reports, 
with a rating of 4.93 (95% CI 4.73 to 5.13; mean differ- 
ence=0.18, 95% CI -0.03 to 0.39, p=0.110; table 3). 

The mean newsworthiness rating of NIHR 
HTA-funded original research was 5.05 (95% CI 4.88 to 
5.22), which was higher than that for non-NIHR HTA 
original research publications, which had a mean news- 
worthiness rating of 4.77 (95% CI 4.75 to 4.79; mean dif- 
ference=0.28, 95% CI 0.03 to 0.53, p=0.03; table 4). The 
mean newsworthiness rating for NIHR HTA-funded ori- 
ginal research reports was similar to publications from 
the same studies publishing outside the HTA Journal, 
which had a mean average newsworthiness rating of 4.96 
(95% CI 4.69 to 5.23; mean difference =0.09, 95% CI 
-0.04 to 0.22, p=0.63; table 4). 

The mean average newsworthiness rating of NIHR 
HTA-funded review reports was 4.64 (95% CI 4.51 to 4.77), 
which was similar to that for non-NIHR HTA-funded 
reviews, which had a mean newsworthiness rating of 4.56 
(95% CI 4.54 to 4.58; mean difference=0.08, 95% CI -0.05 
to 0.21, p=0.24; table 5). The mean average newsworthi- 
ness rating of NIHR HTA-funded review reports was lower 
than that for the same NIHR HTA-funded reviews publish- 
ing outside the HTA Journal, which had an average rating 
of 4.87 (95% CI 4.56 to 5.17; mean differenced. 23, 95% 
CI -0.11 to 0.57, p=0.21; table 5). 



Table 3 Mean average MORE clinical relevance and newsworthiness ratings across publication types 


Number of 
publications 


Average relevance 
(95% CI) 


Average newsworthiness 
(95% CI) 


A: NIHR HTA reports 2007-2012 192 
B: publications 2007—2012 from NIHR HTA 68 

studies included in A 
C: non-NIHR HTA publications 2007-2012 19 921 


5.48 (5.36 to 5.59) 
5.59 (5.42 to 5.75) 

5.32 (5.31 to 5.33) 


4.75 (4.60 to 4.86) 
4.93 (4.73 to 5.13) 

4.70 (4.69 to 4.71) 



HTA,health technology assessment; NIHR, National Institute for Health Research. 



4 



Wright D, Young A, Iserman E, etal. BMJ Open 2014;4:e004556. doi:1 0.1 136/bmjopen-201 3-004556 



e 



Open Access 



Table 4 Mean average clinical relevance and newsworthiness ratings for original research by publication type 


Number of 
publications 


Average relevance 
(95% CI) 


Average newsworthiness 
(95% CI) 


A: NIHR HTA reports 2007-2012 54 
B: publications 2007—2012 from NIHR HTA 46 

studies included in A 
C: non-NIHR HTA publications 2007-2012 13 302 


5.54 (5.34 to 5.74) 
5.57 (5.36 to 5.78) 

5.27 (5.26 to 5.29) 


5.05 (4.88 to 5.22) 
4.96 (4.69 to 5.23) 

4.77 (4.75 to 4.79) 



HTA,health technology assessment; NIHR, National Institute for Health Research. 



NIHR HTA original research reports had a higher 
mean newsworthiness rating than NIHR HTA review 
reports with a mean rating of 5.05 (95% CI 4.88 to 5.22) 
compared with 4.64 (95% CI 4.51 to 4.77; mean differ- 
ence=0.41, 95% CI 0.18 to 0.64, p=0.001). 

DISCUSSION 

This study found that hypothesis one was correct in part: 
NIHR HTA studies reporting in HTA are rated as statis- 
tically more clinically relevant than non-NIHR HTA arti- 
cles publishing in the 120 high-impact journals included 
by McMaster that pass CAP and MORE criteria. 
However, ratings of newsworthiness were similar between 
NIHR HTA studies reporting in HTA and non-NIHR 
HTA publications. The second hypothesis was not 
correct: ratings of clinical relevance and newsworthiness 
were slightly lower for NIHR HTA reports in HTA than 
publications from the same studies publishing outside 
the journal, although these differences were not statistic- 
ally significant. NIHR HTA-funded original research 
reports publishing in the HTA Journal were seen to be 
more newsworthy than NIHR HTA-funded review 
reports. 

The study also found that the proportion of NIHR 
HTA reports and other publications from NIHR 
HTA-funded research passing initial screening criteria 
was higher than that of non-NIHR HTA-funded publica- 
tions (67.8% and 57.6%, compared with 10.6%, respect- 
ively). This provides a further indication of the 
comparatively high relevance and methodological rigour 
of NIHR HTA-funded studies. 

It is interesting to note that subsequent to the comple- 
tion of this study, another project has been completed 
that suggests that the assumptions behind the second 
hypothesis were not correct. Chinnery et at assessed the 



time to publication of projects funded by the NIHR 
HTA Programme publishing in the HTA Journal and as 
a journal article in the wider biomedical literature. 11 
They examined NIHR HTA projects with a planned sub- 
mission date for the draft final report on or before 9 
December 2011 and reported that for primary research 
the median time to publication in the HTA Journal was 
faster than that in other non-HTA Journal publications 
(26.5 and 35.5 months, respectively). This indicates that 
further exploration of the clinical relevance and news- 
worthiness of NIHR HTA reports compared with publi- 
cations from the same studies publishing outside the 
HTA Journal is warranted. 

The strength of this study is that a rigorous, extensive 
and independent approach to assessing the clinical rele- 
vance and newsworthiness of funded research provided 
by MORE was applied to NIHR HTA reports. However, 
there were limitations to the study. First, the study used 
internal research management data that were reliant on 
funded research teams self-reporting their publications. 
It is likely that the number of publications is under- 
reported from NIHR HTA-funded studies and therefore 
some NIHR HTA publications may be included in the 
non-NIHR HTA publication dataset. Second, the study 
compared NIHR HTA reports with non-NIHR HTA pub- 
lications passing CAP and MORE. NIHR HTA-funded 
studies are typically randomised controlled trials, system- 
atic reviews or Technology Assessment Reports (which 
identify, assess and synthesise research evidence from a 
number of healthcare interventions, providing estimates 
of the relative effectiveness and cost effectiveness of a 
range of interventions). While subgroup analyses 
matched original and review research, it could be more 
effective to match publications by specific research 
designs (eg, randomised controlled trials) or by similar 
research hinders. Third, the MORE scales were treated 



Table 5 Mean average clinical relevance and newsworthiness ratings for review research by publication type 


Number of 
publications 


Average relevance 
(95% CI) 


Average newsworthiness 
(95% CI) 


A: NIHR HTA reports 2007-2012 138 
B: publications 2007—2012 from NIHR HTA 22 

studies included in A 
C: non-NIHR HTA publications 2007-2012 6619 


5.46 (5.32 to 5.60) 
5.62 (5.38 to 5.87) 

5.41 (5.39 to 5.43) 


4.64 (4.51 to 4.77) 
4.87 (4.56 to 5.17) 

4.56 (4.54 to 4.58) 



HTA,health technology assessment; NIHR, National Institute for Health Research. 
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as interval scales and therefore mean average ratings 
were obtained. However, the data may be viewed as 
ordinal, in which case median or mode averages would 
be more appropriate. Fourth, raters contributing to 
MORE do so on a voluntary basis and are likely to be 
interested in evidence-based medicine, and therefore 
may not be representative of all physicians. Fifth, no sub- 
group analyses were conducted comparing articles from 
NIHR HTA-funded studies published elsewhere prior to 
publication in the HTA Journal and those published 
after the HTA report. Conducting such subgroup ana- 
lyses will provide interesting data on the impact of prior 
publication on clinical relevance and newsworthiness 
ratings. 

Findings from this study contribute to the relatively 
under-researched area of clinical relevance of health 
research. Commissioning research topics in areas of low 
relevance or in areas where the research evidence is 
already known is a significant contributing factor to 
research waste. 1 The MORE process has previously been 
used to provide exploratory assessments of clinical 
research by research type. McKinlay et al, 6 for example, 
reported that systematic reviews were rated significantly 
higher than original articles for relevance, but signifi- 
cantly lower for newsworthiness. This study reflected 
these findings in part with NIHR HTA-funded original 
research reports having a similar mean average rele- 
vance rating to review reports, but a statistically higher 
mean newsworthiness rating than review reports. The 
lower newsworthiness rating of reviews is to be expected, 
given that all the original articles which inform the 
reviews are already published and thus are in the public 
domain. 6 

The high clinical relevance rating of NIHR HTA 
reports in relation to non-NIHR HTA-funded research 
could be explained by the NIHR HTA Programme's 
identification and prioritisation processes, which are spe- 
cifically designed to ensure that important knowledge 
gaps are identified. Topics are generated via a wide 
range of stakeholder groups and sources, such as web 
submissions, workshops with clinicians, academics and 
patient and public representatives and outputs from the 
James Lind Alliance Priority Setting Partnerships 
(http://www.lindalliance.org/). These are screened by 
specialists and an 'overlap check' is undertaken to 
ensure that there is no duplication of current or com- 
pleted research internally and externally, after which 
topics are developed and prioritised by panels compris- 
ing independent experts and patient and public contri- 
butors. The higher clinical relevance rating of NIHR 
HTA reports may therefore be attributed to the effective- 
ness of these processes. 

An additional factor may be due to the fact that the 
NIHR HTA Programme has its own journal HTA, which 
enables fuller descriptions of research than most other 
journals allow. The accessibility and structure of the 
reports in HTA may be a contributing factor to enhan- 
cing assessments of relevance. However, all NIHR 



HTA-funded studies are contractually obliged to report 
in HTA regardless of the study outcome, resulting in a 
high publication rate of funded studies. 8 The HTA 
Journal is therefore not as likely to be affected by publi- 
cation bias as other health research literature, which 
could therefore limit ratings of relevance and news- 
worthiness when compared to other journals. 

This study presents an exploratory assessment of clin- 
ical relevance and newsworthiness by assessing the 
outputs of one research funder. While this provides an 
initial assessment of relevance and newsworthiness in 
relation to highly rated publications in high-impact jour- 
nals, further research is recommended to confirm the 
observations from the study. In particular, it is recom- 
mended that further analyses are undertaken on outputs 
that are matched by research type (eg, RCTs, systematic 
reviews), research hinders or specific journals. Further 
research should also be conducted on assessing the 
degree to which prior publication of NIHR HTA-funded 
studies influences ratings of clinical relevance and news- 
worthiness of HTA reports. 

Commissioning or publishing research of low-clinical 
relevance is an important factor in the waste of research 
funds. It is recommended, therefore, that careful consid- 
eration is given to clinical relevance when funding and 
disseminating research. 



CONCLUSION 

Research commissioned by the NIHR HTA Programme 
and published in the HTA Journal is more clinically rele- 
vant than highly rated articles publishing in the 120 
journals included in MORE. Reports publishing in the 
HTA Journal had similar ratings of newsworthiness. 
NIHR HTA-funded original research reports had a 
similar mean average relevance rating to review reports, 
but had a significantly higher newsworthiness rating. 

The clear and full accounts provided by the HTA 
Journal and the rigorous topic identification and priori- 
tisation processes adopted by the NIHR HTA 
Programme may be a factor in generating the high clin- 
ical relevance rating. This study illustrates the import- 
ance of considering clinical relevance when 
commissioning and disseminating research in an effort 
to minimise waste in research funding. 

Acknowledgements The authors wish to thank the support of Carole Luke 
and Beata Ferris who assisted in extracting publication records and PubMed 
Identification numbers for National Institute for Health Research (NIHR) 
Health Technology Assessment (HTA)-funded studies. They also thank Dr 
Martin Ashton-Key for his assistance in commenting on an early draft of this 
paper. 

Contributors The study was designed by DW, RMi, RBH, ST and RMa. DW, 
AY and El extracted the data. El conducted the data analyses. DW drafted the 
manuscript. All authors have read and approved the final manuscript. 

Funding This project was funded as part of the NETSCC Research on 
Research Programme. The views and opinions expressed are those of the 
authors and do not necessarily reflect those of the Department of Health and 
of NETSCC. 



6 



Wright D, Young A, Iserman E, etal. BMJ Open 2014;4:e004556. doi:1 0.1 1 36/bmjopen-201 3-004556 



e 



Open Access 



Competing interests DW, AY, RMa and ST are employees at the NIHR 
Evaluation, Trials and Studies Coordinating Centre (NETSCC). RMi is 
employed as the Head of NETSCC and has worked for NETSCC (and its 
predecessor organisation) in senior roles intermittently since 1996. He was an 
editor of the Health Technology Assessment journal (1997-2007) and a 
founder editor for other journals in the new NIHR Journals Library (2011-12). 
RBH and El received payment to McMaster University from NETSCC for time 
taken to analyse the MORE database. 

Provenance and peer review Not commissioned; externally peer reviewed. 

Data sharing statement No additional data are available. 

Open Access This is an Open Access article distributed in accordance with 
the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, 
which permits others to distribute, remix, adapt, build upon this work non- 
commercially, and license their derivative works on different terms, provided 
the original work is properly cited and the use is non-commercial. See: http:// 
creativecommons.org/licenses/by-nc/3.0/ 



REFERENCES 

1 . Chalmers I, Glasziou P. Avoidable waste in the production and 
reporting of research evidence. Lancet 2009;374:86-9. 

2. Gillum LA, Gouveia C, Dorsey ER, et al. NIH disease funding levels 
and burden of disease. PLoS ONE 201 1 ;6:e16837. 



3. Tallon D, Chard J, Dieppe P. Relation between agendas of the 
research community and the research consumer. Lancet 
2000;355:2037-40. 

4. McKibbon K, Wilczynski N, Haynes R. What do evidence-based 
secondary journals tell us about the publication of clinically important 
articles in primary healthcare journals? BMC Med 2004;2:33. 

5. Lokker C, McKibbon KA, McKinlay RJ, et al. Prediction of citation 
counts for clinical articles at two years using data available within 
three weeks of publication: retrospective cohort study. BMJ 
2008;336:655-7. 

6. McKinlay RJ, Cotoi C, Wilczynski NL, etal. Systematic reviews and 
original articles differ in relevance, novelty, and use in an 
evidence-based service for physicians: PLUS project. J Clin 
Epidemiol 2008;61 :449-54. 

7. Raftery J, Powell J. Health technology assessment in the UK. Lancet 
2013;382:1278-85. 

8. Turner S, Wright D, Maeso R, et al. Publication rate for funded 
studies from a major UK health research f under: a cohort study. BMJ 
Open 2013;3:e002521. 

9. Hanney S, Buxton M, Green C, etal. An assessment of the impact 
of the NHS Health Technology Assessment Programme. Health 
Technol Assess 2007;1 1 :iii-iv, ix-xi, 1-180. 

10. Wilczynski N, Morgan D, Haynes RB, etal. The Hedges Team. An 
overview of the design and methods for retrieving high-quality 
studies for clinical care. BMC Med Inform Decis Mak 2005;5:20. 

1 1 . Chinnery F, Young A, Goodman J, et al. Time to publication for 
NIHR HTA programme-funded research: a cohort study. BMJ Open 
2013;3:e004121. 



Wright D, Young A, Iserman E, etal. BMJ Open 2014;4:e004556. doi:1 0.1 1 36/bmjopen-201 3-004556 



7 



